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ABSTRACT 

An  experimental  determination  of  probability  of 
detection  P(D)  as  a function  of  signal  incidence,  SI  (how 
often  a signal  occurs)  showed  that: 

1)  P(D)  decreases  linearly  as  the  SI  is  reduced,  and 

2)  P(D)  remains  finite  as  SI  approaches  zero. 

To  conduct  the  tests,  observers  were  shown  projections  of  film 
strip  photographs  of  the  output  of  TRACOR' s digital  sonar  simu- 
lator. Signals  were  injected  in  10%,  5%,  1%  or  0.17o  of  the 
frames;  noise  alone  was  presented  in  the  other  frames.  At  a 
constant  signal-to-noise  ratio  S/N,  P(D)  decreased  linearly 
from  0.82  at  10%,  SI  to  0.48  at  0.1%,  SI,  at  a constant  false 
alarm  probability  of  0.01.  The  decrease  in  the  detectability 
index  d'  over  this  span  of  SI  was  from  3.21  to  2.28. 
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1.  INTRODUCTION 

An  experimental  study  has  been  made  of  the  effect  of 
signal  incidence  (how  often  a signal  occurs)  on  probability  of 
detection.  This  is  one  of  a series  of  studies  under  Contract 
NObsr-95149,  Mod  2,  Task  2 to  enhance  fleet  utilization  of  the 
AN/SQS-26  sonar  system.  In  conducting  this  study,  advantage  was 
taken  of  TRACOR's  unique  sonar  display  simulation  facility,  its 
pool  of  trained  psychophysical  observers,  and  its  efficient  data 
handling  processes. 

In  contrast  to  most  psychophysical  tests,  in  which 
signals  are  presented  in  a nominal  50%  of  the  trials,  the  signal 
incidences  (SI)  for  this  study  were  10%,  5%,  1%,  and  0.1%.  Equal 
numbers  of  signals  were  presented  at  each  SI  in  order  to  achieve 
the  same  error  bounds  on  the  results  for  each  SI.  The  test 
material  for  this  study  was  a series  of  film  strips  with  signals 
injected  into  a designated  percent  of  the  frames.  SI  is  thus 
expressed  in  percent.  The  signal- to-noise  ratio  (S/N)  was  the 
same  for  all  SI. 

Two  experiments  were  conducted.  In  the  first,  a value 
of  S/N  large  enough  to  yield  a probability  of  detection  P(D)  of 
about  0.5  at  an  SI  of  0.1%  was  used.  At  this  value  of  S/N,  the 
values  of  P(D)  obtained  at  the  various  SI  were 


SI 

1\D) 

10% 

0.820 

5% 

0.675 

1% 

0.530 

0.1% 

0.484 

The  probability  of  false  alarm  P(FA)  was  almost  constant  at  a 
value  of  about  0.01.  The  decrement  in  detectability  index  d' 
in  going  from  10%  to  0.1%  SI  was  from  3.21  to  2.28. 
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In  the  second  experiment,  only  three  values  of  SI 
(10%,  5%  and  1%)  were  studied  at  a lower  value  of  S/N,  in  order 
to  see  if  the  rate  of  decrease  of  P(D)  with  SI  is  constant  as  a 
function  of  S/N.  The  values  obtained  for  P(D)  were  0.514,  0.355, 
and  0.255,  respectively.  A value  of  0.01  was  again  maintained 
for  P(FA) . 

The  procedure  for  conducting  the  tests,  a description 
of  the  test  material  (film  strips)  and  its  manner  of  preparation, 
and  a short  statement  about  the  observers  constitute  Section  2. 

A discussion  of  the  results  is  contained  in  Section  3 and  con- 
clusions are  presented  in  Section  4. 


2.  THE  SIGNAL  INCIDENCE  TEST  PROGRAM 

For  many  years,  there  has  been  concern  about  the  ability 
of  observers  to  detect  weak,  infrequent  signals  (in  a noise  or 
reverberation  background) , whose  occurrence  is  both  temporally 
and  spatially  random,  over  long  periods  of  time^.  Such  a situation 
represents  the  one  faced  by  the  sonar  observer  at  sea  who  must,  in 
addition,  operate  at  a low  probability  of  false  alarm.  The  Navy, 
of  course,  is  interested  in  determining  minimum  detectable  signal 
levels  (MDL),  in  operational  situations.  To  do  this  realistically, 
the  degradation  in  detectability  due  to  rare  signal  occurrence  must 
be  known. 

2.1  TEST  MATERIAL 

2 

An  earlier  study  on  this  contract  demonstrated  the 
feasibility  and  practicality  of  conducting  a study  of  the  effect 
of  signal  incidence  on  an  observer's  probability  of  detection.  It 
was  shown  that  to  obtain  equal  error  bounds  on  the  results  for 
various  SI,  the  total  number  of  signals  presented  at  each  SI  should 
be  the  same.  Obviously,  then,  for  lower  SI,  more  test  material  is 
required . 

The  test  material  for  this  study  was  several  series  of 
film  strips.  Each  of  the  150  frames  of  a film  strip  was  a photo- 
graph of  a cathode  ray  tube  on  which  was  shown  a simulated  six 
echo  cycle  single  beam  history  of  the  AN/SQS-26  signal  processor 
output.  Using  a value  of  0.023  as  the  ± Irj  error  bounds  on  P(D) 
and  16  independent  observations  of  each  film  strip,  the  number  of 
film  strips  required  for  each  SI  was  in  accordance  with  the 
following  schedule: 

^Bergum,  B.O.,  and  Klein,  I.C.,"A  Survey  and  Analysis  of  Vigilance 
Research,"  George  Washington  Univ.,  Human  Resource  Research  Office 
Washington,  D.C.,  Research  Report  8,  Nov.,  1961,  (AD  267  233). 

2 

Young,  J.M. , "The  Feasibility  of  Determining  the  Effect  of  Signal 
Incidence  on  Detectability",  TRACOR  Doc.  No.  67-944-U,  2 Nov.  1967 
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10%  Signal 
5%  Signal 
1%  Signal 
0.17o  Signal 


Incidence 

Incidence 

Incidence 

Incidence 


2 Film  Strips  , 
4 Film  Strips* 
20  Film  Strips , 
200  Film  Strips* 


Figure  1 shows  in  block  diagram  form  the  digital 

simulation  of  the  signal  processor  used  in  conjunction  with 

the  A-scan  display.  The  24-channel  serial  OR-gate  is  used  to 

match  the  display  to  the  signal  processor  so  that  no  more  than 

one  independent  sample  is  placed  in  each  resolvable  location 

on  the  display.  Since  the  24-channel  serial  OR-gate  selects 

and  outputs  to  the  display  only  the  largest  sample  in  each 

sequential  block  of  24,  a loss  in  signal-to-noise  ratio  (S/N) 

results  from  using  this  technique.  It  has  been  shown,  however, 

that  the  OR-gate  loss  is  less  than  that  encountered  if  the  CRT 

is  saturated  with  all  of  the  output  samples  and  used  as  an 
3 

averaging  device  . The  relationship  between  the  S/N  at  the  input 
to  the  24-channel  serial  OR-gate  (signal  processor  output)  and 
the  S/N  at  the  output  is  plotted  in  Figure  2. 

A constant  signal  level  was  used  throughout  the  first 
experiment  such  that  the  average  value  of  S/N  at  the  input  to  the 
display  was  9.3  dB.  For  the  second  experiment,  the  value  used 
was  6.9  dB.  Corresponding  values  at  the  output  of  the  correlator 
are  12.2  dB  and  10.5  dB.  These  latter  values  are  cited  in  the 
remainder  of  this  memorandum.  The  magnitude  of  the  noise  sample 
at  the  time  of  signal  injection  may  have  any  value  under  the  noise 
distribution  curve  shown  in  Figure  3.  There  is  thus  a variation 
of  S/N  from  one  echo  cycle  history  to  the  next  representative  of 
the  ping-to-ping  variation  in  S/N  found  in  recorded  sea  data. 


3 

"Analysis  of  Signal  Processing  and  Related  Topics  Pertaining  to 
the  AN/SQS-26  Sonar  Equipment-A  Summary  Report,  II  (U) ,"  TRACOR 
Document  No.  64-290-C,  p.  181,  October  16,  1964  (CONF). 
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The  intensity  of  each  displayed  mark  was  proportional 
to  its  amplitude.  The  samples  at  the  signal  processor  output 
were  not  thresholded,  i.e,  unity  marking  density  was  employed. 

The  intensity  (brightness)  of  the  CRT  was  adjusted  so  that  the 
largest  samples  did  not  occupy  more  than  one  resolvable  location. 
Only  zero  range  rate  signals  were  used. 

2 .2  OBSERVERS 

The  sixteen  observers  employed  for  the  signal  incidence 
test  were  young  adult  males  with  normal  vision.  Their  ages  ranged 
from  18  to  23  years;  they  had  all  had  prior  experience  as  ob- 
servers on  this  contract^’  . However,  in  all  of  their  earlier 
observational  experience  signals  had  been  present  in  about  50% 
of  the  frames.  In  some  of  the  work"*  a rating  scale  technique 
had  been  employed  which  engendered  lax  decision  criteria  and  a 
subsequent  tendency  to  make  too  many  false  alarms. 

Since  it  was  desired  to  conduct  the  SI  study  under 
conditions  approximating  those  encountered  under  operational 
conditions,  a very  low  probability  of  false  alarm  was  desired. 

In  working  with  psychophysical  subjects  (observers)  however,  it 
is  easy  to  influence  their  behavior  so  that  they  never  make  false 
alarms  or  detect  signals.  The  observers  were  therefore  trained 
to  make  about  1%  false  alarms,  i.e,  in  each  1.50-frame  film  strip 
they  were  allowed,  and  required,  to  make  one  or  two  false  alarms. 

A false  alarm  occurs  when  an  observer  decides  that  a signal  is 
present  when  one  is  not.  Since  each  decision  is  based  on  a single 
observation  of  a six  echo  cycle  history,  the  term  "false  alarm" 
as  used  here  has  a different  significance  from  that  used  by  the 


^Young,  J.  M. , "The  Effect  of  Range  Rate  on  Signal  Detectability," 
TRACOR  Document  No.  67-800-U,  16  October  1967. 

^Young,  J.M. , "Receiver  Operating  Characteristic  (ROC)  Curves  for 
a Simulated  AN/SQS-26  A-Scan  Display,"  TRACOR  Document  No.  67- 
1088-U,  10  January  1968. 
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Navy.  Furthermore,  each  six  echo  cycle  history  is  independent 
of  those  preceding  or  following,  i.e.,  there  is  no  carry-over 
of  data  from  one  frame  to  the  next. 

2.3  PROCEDURE 

The  first  experiment  was  carried  out  in  two  phases. 

Phase  I consisted  of  presentation  to  16  observers  of: 

2 film  strips  at  a SI  of  10% 

4 film  strips  at  a SI  of  5% 

20  film  strips  at  a SI  of  1%, 

Since  Phase  I required  only  some  10  days,  a rather 

quick  indication  was  obtained  that  the  S/N  was  large  enough  to 
overcome  the  expected  degradation  in  probability  of  detection 
P(D)  with  decreasing  SI.  The  goal  was  to  use  a value  of  S/N 
such  that  for  an  SI  of  0.1%  the  P(D)  would  be  about  50%. 

The  film  strips  were  presented  in  a random  manner  with 
respect  to  SI  so  that  the  observers  could  not  anticipate  the 
SI.  The  only  "feedback"  to  the  observers  was  whether  they  were 
maintaining  the  desired  probability  of  false  alarm. 

In  Phase  II,  the  200  film  strips  (30,000  frames),  in 
which  only  30  signals  were  present,  were  viewed  by  the  16  observers. 
Again,  the  signal  occurrence  was  random  (unpredictable).  The  time 
interval  between  signals  ranged  from  as  short  as  10  seconds  (sig- 
nals in  successive  frames)  to  as  long  as  a week.  During  each  two 
hour  observation  period  the  observers  were  presented  600  frames 
of  data  at  the  rate  of  6 frames  per  minute.  After  viewing  each 
frame  they  recorded  their  decision  as  to  whether  or  not  a signal 
had  been  present. 

After  completion  of  the  first  experiment,  it  was  real- 
ized that  comparatively  little  additional  effort  would  be  needed 
to  repeat  the  Phase  I observations  at  a lower  value  of  S/N.  The 
results  should  provide  an  indication  as  to  the  dependence  on  S/N 

S 


n 

. 


of  the  rate  of  decrease  of  P(D)  with  SI.  The  value  of  S/N  for 
this  second  experiment  was  chosen  so  that  for  an  SI  of  10%  a P(D) 
of  about  0.50  would  be  obtained.  It  was  felt  that  the  results 
obtained  from  107,,  5%,  and  1%.  SI  could  be  extrapolated  with 
sufficient  accuracy  to  0.1%  SI. 

The  observations  of  the  26  film  strips  used  in  the 
second  experiment  were  made  in  the  same  manner  as  those  for  the 
first  experiment.  The  observers  were  glad  to  have  more  opportu- 
nities to  detect  signals  than  in  Phase  II  of  the  first  experiment 
even  though  they  soon  became  aware  that  their  ability  to  detect 
signals  was  poorer. 
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3 . RESULTS 

After  completion  of  all  observations  for  a particular 
condition,  the  responses  of  the  16  observers  were  summarized  in 
the  following  manner: 


STIMULUS 


Signal  plus  Noise 

» - ■,  — — — 

Noise  Alone 

Signal 

Number  of  Signals 

Number  of  False 

Present 

Detected,  N(D) 

Alarms,  N(FA) 

No 

Number  of  Signals 

Number  of  Correct 

Signal 

Present 

Missed,  N(M) 

Rejection,  N(CR) 

N(S)  = N(D)  + N(M)  = 
number  of  signals 
presented 

N(N)  = N(FA)  + N(CR)  = 
number  of  noise 
alone  presentations 

The  quantities  of  interest,  P(D)  and  P(FA) , are  found 
by  dividing  N(D)  and  N(FA)  by  N(S)  and  N(N) , respectively.  It  is 
apparent  that  these  two  quantities  completely  describe  the  observers' 
responses . 

The  values  obtained  in  Experiment  1 for  P(D)  and  P(FA) 
are  listed  in  Table  I for  the  various  SI.  The  decrease  in  P(D) 
and  the  almost  constant  value  of  P(FA)  for  decreasing  values  of 
SI  are  obvious.  Also  shown  in  Table  I are  values  of  the  detect- 
ability index,  d'.  This  quantity  is  a measure  of  the  separation 
of  the  mean  values  of  the  noise  (N)  and  signal-plus -noise  (S+ix; 
distributions  along  the  observer's  decision  criterion  axis  (see 
Fig.  4). 
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TABLE  I 

Observers 

1 Results 

for  Experiment  1,  S/N 

= 12.2  dB 

SI,  % 

P (D) 

P(FA)  d' 

’ 

10 

0.820 

0.011  3.21  \ 

1 

5 

0.675 

0.013  2.79  5 

Phase  I 

1 

0.530 

0.008  2.40  ) 

0.1 

0.484 

0.009  2.28  } 

Phase  II 

1 

Table 

II  contains  the  results  obtained 

in  Experiment  2 . 

Since  this  experiment  was 

made  at  a lower  value 

of  S/N,  the 

• 

resulting  values 

of  P (D) 

are  lower. 

TABLE  II 

Observers 

' Results 

for  Experiment  2,  S/N 

= 10.5  dB 

r.i  I 

SI,  % 

P (D) 

P (FA)  d' 

1 

10 

0.514 

0.011  2.31 

5 

0.355 

0.009  1.95 

1 

0.255 

0.012  1.67 

Figures  5 and  6 are  plots  of  P(D)  as  a function  of  SI; 

SI  is  shown  on  logarithmic  and  linear  scales  in  these  plots.  The 
error  bounds  about  the  data  points  in  these  figures  are  the 
theoretical  ± la  limits  based  on  an  assumed  binomial  distribution 
of  the  observers'  responses  in  a "yes -no"  experiment. 

The  nearly  linear  decrease  in  P(D)  with  decreasing  signal 
incidence  shown  in  Fig.  6 was  not  expected  since  a finite  P(D) 
at  zero  signal  incidence  is  the  logical  extension.  This  apparent 
r enigma  is  resolved  by  the  statement:  No  matter  how  rarely  a signal 

* 


12 


n 


CM 


<_> 


o 

n 


co 

ac 


> 

QC 


CO 

co 

o 


o 

CO 

z 

o 


oo 

OC 


l—  CO 
CO  — 
— X 

□ c 


< o 
r — 
ac  a; 

o LU 


o ce 
z c_> 


CL  Z 
CL  O 

<r  — 
_j  co 
CC  — 
CU  <_> 
> LU 
O Q 


J- 

CD 


PROBABILITY  OF  DETECTION,  P(D) 


0.80 


0.60 


0.40 


0.20 


10  5 2 1.0  0.5  0.2  0.1 

SIGNAL  INCIDENCE,  PERCENT 


P(D)  AS  A FUNCTION  OF  SIGNAL  INCIDENCE  FOR  TWO 
VALUES  OF  S/N  AT  THE  OUTPUT  OF  THE  SIGNAL  PROCESSOR. 


14 


ImQtyg [DWG  A6-/3-2428 
4/22/**  YOU  NO  'JK 


PROBABILITY  OF  DETECTION,  P(D) 


m 

¥ 


SIGNAL  INCIDENCE,  PERCENT 


FIG.  6 - P(D)  AS  A FUNCTION  OF  SIGNAL  INCIDENCE  FOR  TWO 
VALUES  OF  S/N  AT  THE  OUTPUT  OF  THE  SIGNAL 
PROCESSOR. 


* 


6500  TRACOR  LANE.  AUSTIN.  TEXAS  78721 

occurs,  the  probability  of  detecting  the  signal  when  it 
does  occur  is  finite  and  depends  on  the  signal -to-noise  ratio. 

A plot  of  d'  as  a function  of  signal  incidence  is  shown 
in  Fig.  7.  A linear  relationship  is  again  exhibited.  In  Fig.  8 
the  reduction  in  d'  is  plotted  as  a function  of  signal  incidence. 

3.1  DISCUSSION  OF  RESULTS 

A straightforward  explanation  of  the  observers*  results 
and  behavior  is  provided  by  the  Theory  of  Signal  Detectability^ 
if  one  assumes  for  ease  of  manipulation  that  the  N and  S + N 
distributions  can  be  represented  on  the  observer's  decision 
criterion  axis  as  Gaussian,  or  normal,  distributions  of  equal 
variance.  Since  more  decisions  by  far  are  made  for  the  N condition, 
an  observer  will  establish  a decision  criterion  that  gives  an 
acceptable  value  of  P(FA).  This  criterion,  C,  in  units  of  standard 
deviation  of  the  N distribution,  can  be  found  from  a tabulation  of 
the  normal  distribution^  if  P(FA)  is  known.  There  is  no  loss  in 
generality  if  the  mean  of  the  N distribution  is  taken  to  be  zero. 

The  value  of  C is  found  from 


[1  - P (FA)]  = v^-  J e * dx  , 

'Leo 

where  the  variance  has  been  normalized  to  unity.  The  values  of 
C obtained  for  the  various  SI  in  Experiments  1 and  2 are  shown  in 
Figures  9 and  10. 

Of  particular  interest  in  these  figures  is  the  decreasing 
separation  of  the  means  of  the  N and  S + N distributions,  which 
indicates  that,  on  the  observers'  decision  criterion  axis,  the 


’Peterson,  W.W.,  T.  G.  Birdsall  and  W.  C.  Fox,  "The  Theory  of  Signal 
Detectability,  ' Transactions  of  the  IRE.  Professional  Group  on 
Information  Theory,  PGIT-4,  p.  171,  September  1954. 

7See,  for  example,  Cramer,  H.,  Mathematical  Methods  of  Statistics. 
Princeton  University  Press,  Table  I,  p"!  557 , 1946 . 


16 


10  9 8 


4 


7 6 5 4 3 

SIGNAL  INCIDENCE,  PERCENT 


FIG.  7 - PLOT  OF  d'  AS  A FUNCTION  OF  SIGNAL  INCIDENCE 
FOR  TWO  S/N  VALUES  AT  THE  OUTPUT  OF  THE 
SIGNAL  PROCESSOR. 
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effective  S/N  is  decreasing  as  SI  decreases.  This  indicates  that 
as  the  SI  decreases  a greater  actual  value  of  S/N  is  required  for 
the  observer  to  discriminate  between  signal-plus-noise  and  noise 
alone  situations  with  the  same  likelihood  of  success.  T' ere  is 
almost  no  change  in  the  position  of  the  observer's  criterion  C for 
the  various  SI. 

Data  for  50%  SI  at  the  same  S/N  are  not  included  in  this 
report  since  it  was  not  felt  to  be  compatible  with  the  other  data. 
The  values  obtained  for  d'  were  but  marginally  greater  than  those 
for  10%  SI;  however  in  most  cases  either  P(FA)  was  too  large 
(greater  than  1%)  or  P(D)  was  less  than  for  10%  SI.  The  50%  SI 
situation  does  not  seem  to  offer  the  same  challenge  to  observers 
as  a low  SI  condition.  Consequently  they  do  not  operate  at 
their  peak  capability. 
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4.  CONCLUSIONS 

On  the  basis  of  the  experiments  reported  here  it  is 
concluded  that: 

1.  The  probability  of  detecting  a signal  at  a constant 

j value  of  S/N  decreases  as  the  signal  incidence  decreases. 

2.  The  rate  of  decrease  of  probability  of  detection  and 
detectability  index  as  signal  incidence  decreases  from  10%  to  0.1% 
is  constant. 

3.  The  probability  of  detection  does  not  approach  zero 

l 

for  very  small  values  of  signal  incidence. 
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